Run both self-hosted stacks on Modal, with a load-test harness - #11

I think these already set in the .env

[major] The Sandbox hard-stops after 24 hours and nothing detects it.

timeout=24 * 60 * 60 is the sandbox's maximum lifetime (and Modal's platform max), after which Modal terminates it, tunnel included. Nothing cleans the modal.Dict when that happens: address and sandbox_id stay populated, so warm streaming_api containers keep dialing the dead tunnel, and cold-started ones pass the if not address guard and boot "successfully" against a corpse. A customer returning after a day sees a green deployment that fails every session (close code 3005 after the API exhausts its connect retries), with no documented cause. Note this is a different failure mode from the Missing expected server metadata keys log the README's Restarting section quotes (that one fires when the endpoint answers but is not the ASR).

Suggested fix: (1) document the 24h lifetime in Deploy/Tear down and cross-link it from the Restarting section as the most likely trigger; (2) in streaming_api, also read sandbox_id and fail fast at startup when modal.Sandbox.from_id(sandbox_id).poll() is not None, raising the same "run start_asr" error; (3) optionally, a background thread in streaming_api that exits the container on sandbox death or address change, which would also remove the manual stop-and-redeploy step the README requires after every start_asr.

[major] Concrete path to resolving Ben's plaintext concern, and the half TLS does not fix.

Confidentiality half: swap unencrypted_ports for encrypted_ports and set AAI_USE_SECURE_CHANNEL_TO_ASR_SERVICE=True. I checked the client side: when that flag is true the streaming API dials the ASR with standard TLS credentials using the default public trust roots, so no vendor change is needed if Modal's TLS tunnel presents a publicly trusted cert on the tunnel hostname. Modal documents that tunnels "terminate TLS automatically" but does not document the CA or whether the socket negotiates ALPN h2 (gRPC requires it), and the comment here says encrypted was never re-tested. One live test settles it: openssl s_client -alpn h2 against the tunnel, then a health check over a secure channel. Worth running before deciding whether streaming ships.

Reachability half: TLS does not authenticate clients. The tunnel address is on the public internet, so anyone who finds it gets free GPU inference on the customer's bill, and since MAX_OPEN_STREAMS=32 is a hard cap, 32 attacker-held streams starve every legitimate session. Complete fix is co-locating the API and ASR in one container so the hop stays on localhost (the Security section already names this; I would promote it to the recommendation), or holding the streaming recipe back until that works.

[major] timeout=3600 caps every WebSocket session at one hour.

Modal treats each WebSocket connection as a single function call ("WebSockets on Modal maintain a single function call per connection"), and the function-level timeout bounds each call. So a realtime session open past 1h is terminated mid-stream. The compose stack never severs sessions (nginx is configured with 10h timeouts). The load tests used 15-20s sessions, so this never surfaced.

Suggested fix: raise timeout here toward Modal's 24h max and document the resulting hard per-session ceiling in the README next to the security caveats (clients must reconnect). Also worth one line on why sync_api keeps 3600 (its requests are bounded by INFERENCE_TIMEOUT_SECONDS=30 anyway).

[major] All four vendor binaries run unsupervised; a post-startup crash black-holes traffic.

Every @modal.web_server body here (both license_proxy functions, streaming_api, sync_api) does a bare subprocess.Popen and returns. Modal only verifies the port opens once at container start; there is no ongoing health check. If the binary exits later (the license proxy is the sharpest case, since the APIs shut themselves down on license failure, and the GPU process can OOM), the container stays alive, keeps counting toward autoscaling capacity, and requests routed to it fail until it happens to scale down. The compose stacks have healthchecks on all four services that catch exactly this.

Suggested fix at all four launch sites: keep the handle and fate-share, e.g.

proc = subprocess.Popen(...)
threading.Thread(target=lambda: (proc.wait(), os._exit(proc.returncode or 1)), daemon=True).start()

so a dead server kills the container and Modal replaces it. This also turns crash-after-bind into a visible restart loop instead of a silent black hole.

[major] "Modal's autoscaler covers the second job" is not true for the component nginx actually balanced.

The ASR is a single Sandbox pinned to one L40S, and Sandboxes do not autoscale; only the CPU streaming_api/license_proxy functions scale. The architecture also cannot grow ASR replicas without reintroducing routing (the modal.Dict holds exactly one address). The X-Model-Version half checks out (the header is routing metadata a single backend safely ignores), but a reader sizing for more concurrent streams and trusting this sentence is misled.

Suggested fix: correct the docstring (autoscaling applies only to the CPU functions) and add one README sentence: this deployment is capped at a single GPU backend; for more capacity run multiple deployments or use compose with nginx.

[major] No read deadline: a live-but-silent server hangs the whole sweep.

for message in ws blocks without a timeout and nothing bounds a session's total duration. The websockets keepalive only unblocks a dead TCP connection; a server that accepts the upgrade and audio but never sends Termination (exactly the overload regime this harness exists to probe, and the PR itself reports connection-level failures at 64) leaves the thread waiting forever. run_level's f.result() has no timeout either, so one stuck session hangs the entire ramp: no table, no exit, at precisely the saturation point the tool is meant to find.

Suggested fix: compute a per-session deadline (e.g. audio_seconds / speed + open_timeout + margin), replace the iterator with ws.recv(timeout=remaining) returning Result(False, ..., detail="session deadline exceeded") on TimeoutError, and optionally pass a timeout to f.result() as a backstop.

[minor] Off-by-one rank: p95 equals max for every level with n <= 20.

Nearest-rank quantile index is ceil(n * p) - 1; int(n * p) is one rank high, so with n <= 20 the printed p95 is always exactly the max (which covers most levels in the documented ramps, and every ramp table in this PR shows it), and with n = 2 the p50 reports the slower of the two as the median. idx = max(0, math.ceil(len(lat) * p) - 1) fixes it.

[minor] For streaming, this metric is tautological: it measures the client's own pacing.

Streaming sessions are realtime-paced by the writer, so wall ~= audio_seconds / speed and audio_x_realtime ~= ok * speed regardless of server capacity. Meanwhile the number that does reflect server health under load, per-session first_turn_s (already collected in Result.extra), is never aggregated; only the first ok result's extra is printed once. Suggest omitting xRT in streaming mode (print "-") or replacing it with first_turn_s percentiles, and scoping the README's saturation guidance ("throughput plateauing while latency climbs") to sync.

[nit] --max-seconds can truncate the audio before the --expect word occurs, turning every request into a false failure that reads like a server problem (the README recommends --max-seconds 20 while --expect defaults to "assemblyai"). A note in the bench README ("pick --expect from the first N seconds when truncating"), or a startup warning when both flags are set with the default expect, avoids the trap.

[nit] Only true for sync. In streaming mode the summary carries first_turn_s and turn count, not words; the character count is what both modes print. (The rest of this README verified clean against the script: the options table matches the argparse definitions exactly, including defaults, and the exit-status claim matches main().)